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ABSTRACT 

Objectives: Appropriate reporting is central to the 
application of findings from research to clinical 
practice. The Strengthening the Reporting of 
Observational Studies in Epidemiology (STROBE) 
recommendations consist of a checklist of 22 items 
that provide guidance on the reporting of cohort, 
case-control and cross-sectional studies, in order to 
facilitate critical appraisal and interpretation of results. 
STROBE was published in October 2007 in several 
journals including The Lancet, BMJ, Annals of Internal 
Medicine and PLoS Medicine. Within the framework of 
the revision of the STROBE recommendations, the 
authors examined the context and circumstances in 
which the STROBE statement was used in the past. 
Design: The authors searched the Web of Science 
database in August 2010 for articles which cited 
STROBE and examined a random sample of 100 
articles using a standardised, piloted data extraction 
form. The use of STROBE in observational studies and 
systematic reviews (including meta-analyses) was 
classified as appropriate or inappropriate. The use of 
STROBE to guide the reporting of observational 
studies was considered appropriate. Inappropriate 
uses included the use of STROBE as a tool to assess 
the methodological quality of studies or as a guideline 
on how to design and conduct studies. 
Results: The authors identified 640 articles that cited 
STROBE. In the random sample of 100 articles, about 
half were observational studies (32%) or systematic 
reviews (19%). Comments, editorials and letters 
accounted for 15%, methodological articles for 8%, 
and recommendations and narrative reviews for 26% 
of articles. Of the 32 observational studies, 26 (81%) 
made appropriate use of STROBE, and three uses 
(10%) were considered inappropriate. Among 19 
systematic reviews, 10 (53%) used STROBE 
inappropriately as a tool to assess study quality. 
Conclusions: The STROBE reporting 
recommendations are frequently used inappropriately 
in systematic reviews and meta-analyses as an 
instrument to assess the methodological quality of 
observational studies. 
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INTRODUCTION 

Appropriate reporting concerns the clear 
and complete presentation of what was 
planned, carried out and found in a partic- 
ular study. Inappropriate reporting makes 



ARTICLE SUMMARY 



Article focus 

■ Appropriate reporting is central for the proper 
application of findings from clinical research into 
clinical practice. 

■ The Strengthening the Reporting of Observa- 
tional Studies in Epidemiology (STROBE) recom- 
mendations aim to provide guidance to authors 
on how to improve the reporting of observational 
studies to facilitate critical appraisal and inter- 
pretation of results. 

■ We examined the reasons for citing STROBE and 
found that most observational studies used 
STROBE as a reporting guideline, while about 
half of systematic reviews used STROBE as a tool 
to assess the methodological quality of the 
studies. m 

Key messages 

■ Our study provides further evidence that authors 
of systematic reviews inappropriately use 
reporting guidelines to assess methodological 
study quality. Given the identified common 
misuse of STROBE, we discuss possible 
reasons and potential pitfalls of such misuse. ^ 

Strengtlis and limitations of this study 

■ We conducted a systematic review of the 
literature to address a relevant and insufficiently 
discussed issue concerning misuses of reporting 
guidelines. One of the main concerns of such 
misuse is the potential introduction of bias into 
systematic reviews and meta-analysis. 

■ A limitation of our findings is the fact that we 
included only articles which cited STROBE. This 
may have resulted in a selection bias, since some 
researchers may use STROBE in their study and 
mention it in their manuscript but do not formally 
cite it. 



the sound application of study findings into 
clinical practice difficult, if not impossible. 
For instance, clinicians are often faced with 
reports of studies which do not provide 
enough detail of interventions for them to be 
implemented in clinical practice.^ Inade- 
quate reporting may also be a problem for 
the synthesis of evidence. It is common for 
systematic reviewers or meta-analysts to 
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exclude otherwise eligible studies owing to incomplete 
reporting. Given the human and financial resources 
needed to conduct clinical studies, it is clear that inade- 
quate reporting has also important ethical and moral 
implications.^ 

To help improve the reporting of key items, reporting 
guidelines have been developed for various types of 
research design. The positive effect of reporting guide- 
lines on the quality of reporting has been documented 
in several reviews.'^ The Strengthening the Reporting 
of Observational Studies in Epidemiology (STROBE) 
Statement was developed in a collaborative effort of 
epidemiologists, methodologists, statisticians, researchers 
and journal editors to improve the reporting of obser- 
vational studies, including cohort, case— control and 
cross-sectional studies. The initiative was established in 
2004 and subsequently developed in several meetings. 
STROBE was published in October 2007 simultaneously 
in several journals including leading biomedical jour- 
nals such as The Lancet, BMJ, Annals of Internal Medicine 
and PLoS Medicine. The STROBE recommendations 
are accompanied by an explanation and elaboration 
document (E&E) which explains in detail the rationale 
of each item and provides examples of transparent 
reporting from published articles. To date, the STROBE 
Statement is endorsed by over 100 journals as well as by 
the International Committee of Medical Journal Editors.*^ 

The purpose of the STROBE Statement 'is to ensure 
clear presentation of what was planned, done, and found 
in an observational study' and the authors of STROBE 
clearly stress that 'the recommendations are not 
prescriptions for setting up or conducting studies, nor do 
they dictate methodology or mandate a uniform presen- 
tation.'^ Notwithstanding the clear statement of the 
purpose of STROBE by its authors, some journal editors 
are concerned that the STROBE recommendations may 
be inappropriately used as an assessment tool to judge 
study quality or that researchers may use STROBE as 
a guideline to set up or conduct observational studies.^" 

In the present study, we examined in which circum- 
stances and context STROBE has been cited by the 
researchers (how, where and why) , and the background 
of researchers who cited STROBE. We considered the 
use of STROBE other than in relation to its original 
purpose as a reporting guideline for observational 
studies. Our results were presented and discussed during 
the August 2010 STROBE group meeting, when the 
group met to discuss a possible revision of the STROBE 
recommendations in light of the group's experience, 
anecdotal reports, feedback from STROBE users and 
new evidence. 

METHODS 
Literature searcli 

On 12 August 2010, we conducted a two-step literature 
search in the Web of Knowledge database (http://www. 
isiknowledge.com/). The first step consisted of a search 
for STROBE publications. Our search strategy to identify 



STROBE publications contained words used in the title 
of the STROBE article and author name (appendix 1). 
This first search was conducted in order to set up the 
second search, where we identified articles which had 
cited any of the STROBE publications identified in the 
first step, using the 'create citation report' tool available 
in the Web of Knowledge database. 

Article selection 

Eligibility criteria for our first search (ie, search for 
STROBE publications) consisted of any STROBE publi- 
cation regardless of language or version of publication 
(ie, checklist or E&E). We excluded the Strengthening 
the Reporting of Genetic Association Studies (STREGA) 
recommendations, which is an extension of STROBE to 
genetic epidemiology. 

There were no eligibility restrictions for our second 
search (ie, search of articles which cited STROBE). Any 
publication type, such as comments, editorials, system- 
atic reviews or observational research, was eligible for 
inclusion. We then randomly selected a sample of 100 
articles from the final list of articles which cited STROBE 
for detailed assessment. 

Data collection 

We collected the following information for STROBE 
publications identified through our initial search: 
journal in which STROBE was published, language of 
publication and number of citations which STROBE 
received per year. 

For the data extraction from articles which cited 
STROBE, a data-extraction form was developed by the 
authors and piloted by two reviewers (BRdC and MC) 
on a sample of 40 studies which were not part of the 
randomly selected articles. A single reviewer (BRdC) 
then conducted data extraction on a randomly selected 
sample of 100 articles for the following items: verbatim 
text of sentence (s) which included STROBE citation; 
reason for citation; STROBE article cited; type of article 
that cited STROBE (observational research; systematic 
review or meta-analysis; comment, editorial or letter to 
the editor; methodological article; or recommendations 
or narrative review); and background of researchers. 
Categories for 'reason of citation' were defined during 
the piloting of the data-extraction form, and additional 
categories were defined during data extraction as 
necessary. Whenever the data extractor (BRdC) was in 
doubt about the classification of an article, he discussed 
it with other authors until a decision was reached. In case 
of disagreements, the final decision was made by the 
senior epidemiologist (ME). 

Classification of reasons for citation 

Results of our search for articles which cited STROBE are 
presented narratively. Our sample of 100 randomly 
selected articles were classified into eight groups 
according to reason for citation: STROBE used as 
a reporting guideline; STROBE used as a methodological 
(ie, research design or conduct) guideline; STROBE used 
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as an assessment tool of reporting quality; STROBE used 
as an assessment tool of methodological quality; STROBE 
cited to stress the importance of reporting guidelines; 
'unclear'; and 'other.' 

We considered the use of STROBE in reports of 
observational research to be appropriate when STROBE 
was used as a reporting guideline to ensure a clear and 
complete report of the study's design, conduct and 
findings. In this case, it was clear from the text that one 
or more items of STROBE were used to guide the 
reporting of the study. Our definition of appropriate use 
of STROBE is in line with the stated intentions of its 
authors 'solely to provide guidance on how to report 
research well' and 'not prescriptions for designing or 
conducting studies.'^ Accordingly, we considered inap- 
propriate the use of STROBE as a methodological 
guideline (ie, guideline to design or conduct observa- 
tional research or as an assessment tool of methodo- 
logical quality of publications reporting observational 
research). An example of what we considered inappro- 
priate is the use of STROBE to assign an overall score 
of the methodological quality of a study. As stressed 
by the authors, STROBE was not developed to explain 
how research should be done, and so cannot validly 
be used to assess methodological quality. In box I, 
we present text examples of appropriate and inappro- 
priate uses of STROBE. The remainder of the citations 
which did not fall into either category were considered 
neutral, that is, neither appropriate nor inappropriate. 
Only observational research, systematic reviews or 
meta-analyses could be classified as appropriate or 
inappropriate. 

For exploratory purposes, we classified articles which 
appropriately and inappropriately used STROBE 
according to the affiliation of the authors as a proxy for 
their background. Authors affiliated with epidemiology 
or public-health departments were assumed to have 
methodological training in epidemiological research. 
Owing to the identified large proportion of systematic 
reviews which inappropriately cited STROBE, we exam- 
ined these reviews further, extracting information on the 
language of publication (English or other) and type of 
journal (specialist or general medical journal). 

RESULTS 

Search for STROBE publications 

We identified 16 STROBE publications. Eleven of these 
publications were published in English, three in Spanish 
and two in German. Ten of these publications were 
published in 2007, five in 2008 and one in 2009. Table 1 
displays the journals where STROBE was published, the 
number of citations received, the impact factor of the 
journal, the language of the publication and the date of 
the publication. 

Searcli for articles which cited STROBE 

We identified 643 citations of STROBE since its 
first publication in October 2007 until August 2010. 



Box 1 Text examples of appropriate and inappropriate 
use of Strengtliening tine Reporting of Observational 
Studies in Epidemiology (STROBE) 



Based on the original purpose of STROBE, we consider the 
following text examples as appropriate or inappropriate use 
of STROBE ^ 

Appropriate 

■ 'The reporting of this study conforms to the STROBE 
statement.'''^ 

■ 'The STROBE guidelines were used to ensure the 
reporting of this observational study. '^^ 

■ 'The paper is reported following the STROBE state- 
ment.'''^ — 



Inappropriate 

■ 'This prospective study was designed following recom- 
mendation of the STROBE statement.'^'* 

■ 'The analysis followed the principles of the Strengthening 
the Reporting of Observational Studies in Epidemiology 
initiative. '^^ 

■ 'The quality of selected studies was assessed using 
a modified version of the STROBE statement. ...The 
STROBE was modified by adding questions about the 
serologic method used to confirm dengue diagnosis, use 
of viral isolation, and whether the study was based on 
a single dengue outbreak or transmission season. Use of 
viral isolation increased the score whereas single 
outbreak studies received no additional points. The 
quality score was the number of items from the 
STROBE checklist addressed as a percentage of the 
total number of items applicable (minimum of 23 and 
maximum of 25). Studies with a quality assessment 
below 50% were excluded. '^^ 



Two-hundred and four citations (32%) concerned the 
STROBE E&E document, and 439 (68%) citations 
concerned the STROBE checklist. The STROBE check- 
list published in The Lancet was the most cited, receiving 
110 (17%) citations. The number of citations was asso- 
ciated with the impact factor of the journal as shown in 
table 1, the main exception being the Journal of Clinical 
Epidemiology, which received a large number of citations 
despite a lower journal impact factor. 

Half of the articles in our sample of 100 randomly 
selected articles were observational studies (32%) or 
systematic reviews/meta-analyses (19%). Table 2 displays 
the characteristics of these 51 research articles identified 
in our random sample. 

Comments, editorials and letters accounted for 15% of 
the articles, methodological articles for 8%, and recom- 
mendations and narrative reviews for 26%. Of the 32 
observational studies assessed, 26 (81%) made an 
appropriate use of STROBE, and three (10%) uses were 
considered inappropriate. Of the 19 systematic reviews 
or meta-analyses assessed, 10 (53%) made an inappro- 
priate use of STROBE. Thus, of the 51 research articles, 
26 (51%) used STROBE appropriately, and 13 (25%) 
used STROBE inappropriately. 
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Table 1 Overview of Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) publications ordered 
by amount of citations received 



STROBE publication (journals) 


No of citations 


Impact factor* 


Language 


Date of publication 


The Lancet 


110 


30.8 


English 


October 2007 _ 


Ann Intern Med 


99 


16.2 


English 


October 2007 M 


Ann Intern Med (E&E)t 


97 


16.2 


English 


October 2007 ^ 


BMJ 


69 


13.7 


English 


October 2007 H 


Plos Med (E&E)t 


62 


13.1 


English 


October 2007 ■ 


J Clin Epidemiol 


51 


3.0 


English 


October 2008 ■ 


Plos Med 


43 


13.1 


English 


October 2007 \ 


Epidemiology (E&E)t 


42 


5.5 


English 


November 2007 


Epidemiology 


21 


5.5 


English 


November 2007 m 


Prev Med 


15 


3.2 


English 


October 2007 fl 


Bull World Health Organ 


14 


5.3 


English 


November 2007 1 


Gaceta Sanitaria 


7 


1.2 


Spanish 


March 2008 1 


Internist 


4 


0.3 


German 


June 2008 1 


Gaceta Sanitaria (E&E)t 


3 


1.2 


Spanish 


March 2009 M 


Rev Esp Salud Publica 


3 




Spanish 


May 2008 ^ 


Notfall & Rettungsmedizin 


0 


0.6 


German 


May 2008 



*lmpact factors retrieved from 2009 Journal Citation Report Science Edition. 

t Publication of the explanation and elaboration (E&E) document that accompanied the STROBE recommendations. 



Finally, out of the 26 studies which used STROBE 
appropriately, 13 (50%) had authors affiliated with 
epidemiology or public-health departments, compared 
with six (46%) among the 13 studies with inappropriate 
use of STROBE. As for the 10 systematic reviews that 
inappropriately used STROBE as a tool to assess study 
quality, none were published in the Cochrane Database 
of Systematic Reviews, nine were published in medical 
specialist journals, five had at least one of the authors 
affiliated with epidemiology or public health depart- 
ments, and all were published in English. The 10 arti- 
cles related to cardiology, endocrinology, psychology, 
neurology, paediatrics, ophthalmology, internal medi- 
cine, periodontology, pharmacology and infectious 
diseases. 

DISCUSSION 

STROBE has been cited over 600 times since its publi- 
cation in 2007. Our citation analysis showed that most of 
the observational studies used STROBE as a guideline 



for reporting, whereas about half of the systematic 
reviews and meta-analyses used STROBE inappropri- 
ately, as a methodological quality-assessment tool. 

The inappropriate interchangeable use of the terms 
'methodological quality' and 'reporting quality' is 
common and may explain why some researchers used 
STROBE as a tool for the assessment of methodological 
quality or as a guideline to design and conduct obser- 
vational research. Reporting quality refers to the 
completeness with which a study is presented and 
whether major items for the proper appraisal of internal 

17 

and external validity of findings are clearly reported. 
Methodological quality refers to the appropriateness 
of the methods employed in the design and conduct 
of epidemiological research, which determines the 
reliability of findings (ie, internal validity). 

Even though the authors of STROBE clearly state that 
its purpose is to guide reporting of observational 
research, we found that it is commonly used as a meth- 
odological quality-assessment tool in systematic reviews 



Table 2 Characteristics of 51 classifiable articles citing Strengthening the Reporting of Observational Studies in Epidemiology 
(STROBE) 



Classification 


Reason for citation 


Observational 
research 


Systematic review, 
meta-analysis 


Appropriate 


Guideline for reporting of study 


26 


0 


Inappropriate 


Guideline for design and conduct of study 


3 






Tool to assess methodological quality 


0 




Neutral 


Tool to assess reporting quality 


0 


i 




Example to stress the importance of reporting guidelines 


0 




1 


Other* 


2 






Unclear 


1 






Total 


32 


19 



*This category includes observational studies which cited STROBE as a guideline without making it clear whether the guidance related to 
reporting or methodology, or articles which mentioned STROBE in a commentary. g 
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and sometimes as a guideline to design and conduct 
observational studies. Some journal editors have 
expressed concern that the STROBE recommendations 
may be inappropriately used as an assessment tool to 
judge study quality, or that researchers may use STROBE 
as a guideline to set up or conduct observational 
studies.^** Our results show that these concerns were 
justified, although we feel that some items of STROBE 
might be useful when designing or conducting an 
observational study. In particular, the explanation and 
elaboration STROBE document (E&E) could be useful 
to inform methodological decisions, especially for 
researchers with little formal training in epidemiology. 

Although some items of STROBE may be related to 
risk of bias, many of the items are exclusively related to 
transparent reporting. For instance, the first item of 
STROBE asks researchers to indicate in the title of their 
manuscript the design of their study. Obviously, whether 
authors report the design of their study in the title of 
their manuscript will not influence the effect estimates 
of their study or compromise the generalisability of their 
findings. To illustrate the pitfalls of using STROBE as 
a methodological assessment tool, we refer to the last 
text example we provided as an example of inappro- 
priate use of STROBE in box 1. In this example, the 
authors of a systematic review transformed STROBE into 
a scale by assigning scores to each of its items, and a final 
score was used to determine the eligibility of studies for 
inclusion into the systematic review. By doing so, authors 
may have introduced bias to their findings, as different 
tools used to score methodological quality may result in 
different conclusions regarding the overall appropriate- 
ness of the methods employed in a particular study. 

The misuse of STROBE as a tool to assess methodo- 
logical quality may be explained by the lack of validated 
and accepted tools for such assessments. As a conse- 
quence, authors who want to assess methodological 
quality of studies may turn to reporting guidelines. The 
absence of reliable tools that are based on sound 
empirical evidence to assess the quality of observational 
studies was documented by Sanderson et a/ in 2007.^^ 

There is an extensive literature indicating that the 
misuse of findings of scientific articles is common across 
different fields of healthcare.^ In a similar analysis, 

Moher et al reported that the Consolidated Standards of 
Reporting Trials statement, a reporting guideline for 
randomised controlled trials, has also been misused by 
4—10% of assessed articles as a guideline for quality 
assessment.'^ Taken together, these results indicate 
a failure of not only authors but also peer-reviewers in 
recognising and avoiding misuse of guidelines. The peer- 
review process should ideally prevent misuses that are of 
major concern, in particular when misuses may dictate 
the outcomes of research as discussed above. 

We observed an association between the numbers of 
citations received and the impact factor of the journal in 
which STROBE was published. Even though the 
contents of many STROBE publications are identical. 



researchers preferred to cite a STROBE article published 
in a journal with a higher impact factor. This association 
was also observed by Perneger, who concluded that 
citations received by an article are not merely explained 
by scientific merit.'^^ Although reasons directing authors' 
choices are not clear, for some, a high impact factor may 
suggest the acceptance by a broader audience. The main 
exception to this association was the STROBE checklist 
published by the Journal of Clinical Epidemiology, which 
was also highly cited. This may have occurred because 
STROBE is particularly relevant for the daily work of the 
readers of this journal. 

We found no clear pattern regarding affiliations of 
authors to an institution with a methodological focus. 
In particular, we found no difference in affiliations 
between authors who used STROBE appropriately and 
authors who used STROBE inappropriately. Similarly, no 
clear patterns emerged when we examined more closely 
the characteristics of the 10 systematic reviews that 
inappropriately cited STROBE. 

A limitation of our findings is the fact that we included 
only articles which cited STROBE. This may have 
resulted in selection bias, since some researchers may 
have used STROBE for their study but not formally cited 
it. Such studies could not be identified by our search 
strategy. However, it is unclear to us in which direction 
this possible selection bias may have influenced our 
findings. 

Citations to the STROBE checklist have been steeply 
increasing since its first publication. STROBE is 
commonly used according to its original purpose, that is, 
a reporting guideline of observational research. 
However, despite the clear statement of its objective by 
its authors, STROBE is misused by some researchers 
reporting observational studies and about half of authors 
reporting systematic reviews, because they used it either 
as a tool to assess methodological quality or as a guide- 
line on how to design and conduct observational studies. 
Further studies are required to define the consequences 
of the inappropriate use of reporting recommendations 
in clinical and epidemiological research, in particular 
the use of these checklists as an instrument to assess the 
methodological quality of studies. 
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APPENDIX 1 

Search strategy for identifying STROBE articles 



Step no 


Search strategy* 


1 


Title=(Strengthening the Reporting of 




Observational Studies in Epidemiology) AND 




Author=(von Elm) 


2 


Title=(Strengthening the Reporting of 




Observational Studies in Epidemiology) AND 


I3 


Author=(Vandenbroucke) 




#1 OR #2 


4 


Title=(STREGA) 


5 


#3 NOT #4 ^ 


*Web of Science was searched through the ISI Web of Knowledge platform 
(http://isiknowledge.com/). The search was conducted on 12 August 2010. 
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